{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 背景介绍\n",
    "## 强化学习\n",
    "- 强化学习（英语：Reinforcement learning，简称RL）是机器学习中的一个领域，强调如何基于环境而行动，以取得最大化的预期利益。\n",
    "- 核心思想：智能体`agent`在环境`environment`中学习，根据环境的状态`state`（或观测到的`observation`），执行动作`action`，并根据环境的反馈`reward`（奖励）来指导更好的动作。\n",
    "\n",
    "比如本项目的Catcher接球小游戏中，`agent`就是动图最下方的小白板，它有向左向右两种`action`，如果接到红色方块就能获得`reward`：\n",
    "\n",
    "![file](https://ai-studio-static-online.cdn.bcebos.com/82e0622a9e9b49daa470f89d63555556a8760876c22f429a9abb7daf573047ce)\n",
    "\n",
    "## Sarsa \n",
    "\n",
    "- `Sarsa`全称是`state-action-reward-state'-action'`，目的是学习特定的`state`下，特定`action`的价值`Q`，最终建立和优化一个`Q`表格，以`state`为行，`action`为列，根据与环境交互得到的`reward`来更新`Q`表格，更新公式为：\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/776b473b7f994702a3e05c5eac1156a7ce03b9e6bdb5453085fa9cbb86979715)\n",
    "\n",
    "- `Sarsa`在训练中为了更好的探索环境，采用`ε-greedy`方式来训练，有一定概率随机选择动作输出。\n",
    "\n",
    "## Q-Learning\n",
    "\n",
    "`DQN`本质上是在`Q`网络上的更新，所以这里先复习`Q-Learning`这种策略选择方式的知识点。\n",
    "\n",
    "`Q-Learning`也是采用`Q`表格的方式存储`Q`值（状态动作价值），决策部分与`Sarsa`是一样的，采用`ε-greedy`方式增加探索。\n",
    "\n",
    "区别在于\n",
    "\n",
    "- `Sarsa`是`on-policy`的更新方式，先做出动作再更新。\n",
    "- `Q-Learning`是`off-policy`的更新方式，更新`learn()`时无需获取下一步实际做出的动作`next_action`，并假设下一步动作是取最大`Q`值的动作。\n",
    "\n",
    "`Q-Learning`的更新公式是：\n",
    "\n",
    "![file](https://ai-studio-static-online.cdn.bcebos.com/38158582039041edad0a5a704ba792d0e464f2eb8a394577bf88051cc52d6b66)\n",
    "\n",
    "![file](https://ai-studio-static-online.cdn.bcebos.com/7e698793edc34e918422952541c80ca6511b0b83f6c54ad1a5187765475de142)\n",
    "\n",
    "但是使用`Q`表格来存储每一个状态`state`, 和在这个`state`每个行为`action`所拥有的`Q`值存在瓶颈——问题太复杂，状态特别多，不仅训练耗时，内存也装不下。\n",
    "\n",
    "相关资料：\n",
    "- 不可错过的精彩课程 [强化学习7日打卡营](https://aistudio.baidu.com/aistudio/education/group/info/1335?activityId&directly=1&shared=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# DQN\n",
    "## DQN的两大利器\n",
    "\n",
    "再回顾下`DQN`（Deep Q Network）的原理，上文已经说到，本质上`DQN`还是一个`Q-learning`算法，更新方式一致。为了更好的探索环境，同样的也采用`ε-greedy`方法训练。\n",
    "\n",
    "![file](https://ai-studio-static-online.cdn.bcebos.com/769b5365963d4790ad879d4c0ed5e72398de941ff2404c3cb2ab89d3c45701ad)\n",
    "\n",
    "但是`DQN`论文引入了卷积神经网络，并提出两大利器，用于解决DL和RL结合时遇到的问题\n",
    "\n",
    "- 经验回放 `Experience Replay`：主要解决样本关联性和利用效率的问题。使用一个经验池存储多条经验`s,a,r,s'`，再从中随机抽取一批数据送去训练。\n",
    "- 固定Q目标 `Fixed-Q-Target`：主要解决算法训练不稳定的问题。复制一个和原来`Q`网络结构一样的`Target Q`网络，用于计算`Q`目标值。\n",
    "\n",
    "### experience replay经验回放\n",
    "\n",
    "经验池中的记忆库用来学习之前的经历，又因为`Q-learning`是一种 `off-policy`离线学习法, 它能学习当前经历着的, 也能学习过去经历过的, 甚至是学习别人的经历， 所以每次`DQN`更新的时候, 我们都可以随机抽取一些之前的经历进行学习。\n",
    "\n",
    "为什么要随机抽取？因为状态的转移是连续的，四元组`s,a,r,s'`如果直接按顺序取一批四元组作为训练集，那么是容易过拟合的，因为训练样本间不是独立的！因此需要从经验池中随机抽取少量四元组作为一个`batch`，这样既保证了训练样本是**独立同分布**的，也使得每个`batch`**样本量不大**，能加快训练速度。\n",
    "\n",
    "总结一下，随机抽取这种做法打乱了经历之间的相关性, 也使得神经网络更新更有效率。    \n",
    "\n",
    "### Fixed-Q-Target目标网络\n",
    "\n",
    "`Fixed-Q-Target`的作用其实也是一种打乱相关性的机制，使用`Fixed-Q-Target`会使得`DQN`中出现两个结构完全相同但是参数却不同的网络，预测`Q`估计的的网络`MainNet`使用的是最新的参数，而预测`Q`现实的神经网络`TargetNet`参数使用的却是很久之前的。\n",
    "\n",
    "这也是为了防止过拟合。试想如果只有一个神经网络，那么它就在会不停地更新，那么它所追求的目标是在一直改变的，即在神经网络的参数改变的时候，不止`Q(s, a)`变了，`maxQ(s’, a’)`也变了。这样的好处是一段时间里目标`Q`值使保持不变，一定程度降低了当前`Q`值和目标`Q`值的相关性，提高了算法稳定性。\n",
    "\n",
    "## DQN算法具体流程\n",
    "\n",
    "![file](https://ai-studio-static-online.cdn.bcebos.com/f2aa1199a0cf4a31a68061a03b82f9f5a0bd5c98869c4daca5e735b2e98cf4e0)\n",
    "\n",
    "\n",
    "## DQN存在的过估计（over estimate）问题\n",
    "\n",
    "`DQN`直接选取目标网络（Target Q Network）中下一个`state`各个`action`对应的`Q`值最大的那一个`Q`值，并且会不断向下传递，导致`DQN`估计的`Q`值往往会偏大。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## PARL框架DQN代码分析"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'PARL'...\n",
      "remote: Enumerating objects: 4066, done.\u001b[K\n",
      "remote: Counting objects: 100% (4066/4066), done.\u001b[K\n",
      "remote: Compressing objects: 100% (2059/2059), done.\u001b[K\n",
      "remote: Total 4066 (delta 2543), reused 2959 (delta 1837), pack-reused 0\u001b[K\n",
      "Receiving objects: 100% (4066/4066), 31.02 MiB | 1.36 MiB/s, done.\n",
      "Resolving deltas: 100% (2543/2543), done.\n",
      "Checking connectivity... done.\n"
     ]
    }
   ],
   "source": [
    "# 拉取工具库\n",
    "!git clone https://gitee.com/paddlepaddle/PARL.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!pip install gym -i https://pypi.tuna.tsinghua.edu.cn/simple\n",
    "!pip install atari-py -i https://pypi.tuna.tsinghua.edu.cn/simple # 玩Gym的Atari游戏必装依赖，本次作业使用了Atari的Pong(乒乓球)环境\n",
    "!pip install tqdm -i https://pypi.tuna.tsinghua.edu.cn/simple\n",
    "!pip install parl==1.3.1 -i https://pypi.tuna.tsinghua.edu.cn/simple"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "PARL中`DQN`的实现可以在`parl/algorithms/fluid/dqn.py`文件找到"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class DQN(Algorithm):\n",
    "    def __init__(self, model, act_dim=None, gamma=None, lr=None):\n",
    "        \"\"\" DQN algorithm\n",
    "        \n",
    "        Args:\n",
    "            model (parl.Model): model defining forward network of Q function | 定义Q函数的前向网络结构\n",
    "            act_dim (int): dimension of the action space | action空间的维度，即有几个action\n",
    "            gamma (float): discounted factor for reward computation. | reward的衰减因子\n",
    "            lr (float): learning rate. | 学习率\n",
    "        \"\"\"\n",
    "        self.model = model\n",
    "        self.target_model = copy.deepcopy(model)\n",
    "\n",
    "        assert isinstance(act_dim, int)\n",
    "        assert isinstance(gamma, float)\n",
    "\n",
    "        self.act_dim = act_dim\n",
    "        self.gamma = gamma\n",
    "        self.lr = lr\n",
    "\n",
    "    def predict(self, obs):\n",
    "        \"\"\" \n",
    "        use value model self.model to predict the action value\n",
    "        使用self.model的value网络来获取 [Q(s,a1),Q(s,a2),...]\n",
    "        \"\"\"\n",
    "        return self.model.value(obs)\n",
    "\n",
    "    def learn(self,\n",
    "              obs,\n",
    "              action,\n",
    "              reward,\n",
    "              next_obs,\n",
    "              terminal,\n",
    "              learning_rate=None):\n",
    "        \"\"\" \n",
    "        update value model self.model with DQN algorithm        \n",
    "        使用DQN算法更新self.model的value网络\n",
    "        \"\"\"\n",
    "        # Support the modification of learning_rate\n",
    "        if learning_rate is None:\n",
    "            assert isinstance(\n",
    "                self.lr,\n",
    "                float), \"Please set the learning rate of DQN in initializaion.\"\n",
    "            learning_rate = self.lr\n",
    "\n",
    "        pred_value = self.model.value(obs)\n",
    "        next_pred_value = self.target_model.value(next_obs)\n",
    "        best_v = layers.reduce_max(next_pred_value, dim=1)\n",
    "        best_v.stop_gradient = True\n",
    "        target = reward + (\n",
    "            1.0 - layers.cast(terminal, dtype='float32')) * self.gamma * best_v\n",
    "\n",
    "        action_onehot = layers.one_hot(action, self.act_dim)\n",
    "        action_onehot = layers.cast(action_onehot, dtype='float32')\n",
    "        pred_action_value = layers.reduce_sum(\n",
    "            layers.elementwise_mul(action_onehot, pred_value), dim=1)\n",
    "        cost = layers.square_error_cost(pred_action_value, target)\n",
    "        cost = layers.reduce_mean(cost)\n",
    "        optimizer = fluid.optimizer.Adam(\n",
    "            learning_rate=learning_rate, epsilon=1e-3)\n",
    "        optimizer.minimize(cost)\n",
    "        return cost\n",
    "\n",
    "    def sync_target(self):\n",
    "        \"\"\" \n",
    "        sync weights of self.model to self.target_model\n",
    "        把 self.model 的模型参数值同步到 self.target_model\n",
    "        \"\"\"\n",
    "        self.model.sync_weights_to(self.target_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## DQN解决Pong\n",
    "PARL的`DQN_variant`解决`Pong`示例项目中，默认使用`DQN`进行训练，可以在VisualDL中观察训练过程，下图为本地训练效果：\n",
    "![file](https://ai-studio-static-online.cdn.bcebos.com/419f70c0da024692b9744b3275981798e24896088d21492e88b64b3efe898126)\n",
    "![file](https://ai-studio-static-online.cdn.bcebos.com/c9b57570364d4f27b652bf914ff00aacaddb62909a1b4de1964a8d151ec45844)\n",
    ">  注：AI Studio中查看VisualDL效果，请参考[VisualDL2.0--基于「手写数字识别」模型的全功能展示](https://aistudio.baidu.com/aistudio/projectdetail/622772)等项目"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/aistudio/PARL/examples/DQN_variant\n"
     ]
    }
   ],
   "source": [
    "!cd PARL/examples/DQN_variant && python -u train.py --rom ./rom_files/pong.bin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Double DQN\n",
    "\n",
    "`Double DQN`是`DQN`的一种改进，旨在解决DQN训练过程中存在的过估计（over estimate）问题。在训练过程中，`Double DQN`首先使用预测网络（Predict Q Network）计算下一个`state`的对应各个`action`的`Q`值，然后选取最大的那个`Q`值对应`action`的索引，再使用目标网络计算该状态的对应各个状态的`Q`值，然后选取预测网络中给定`action`索引对应的`Q`值，但是它可能不是最大的那个，从而一定程度上避免了过度估计，提高了训练`DQN`的稳定性和速度。\n",
    "\n",
    "这是一种相互督查的方法。如果只有一个`Q`网络，经常会过度估计。那就用两个`Q`网络，因为两个`Q`网络的参数有差别，所以对于同一个动作的评估也会有少许不同。我们选取评估出来较小的值来计算目标。这样就能避免`Q`网络过度估计的情况发生了。\n",
    "\n",
    "实验结果也证明，`Double DQN`有效地防止了过度估计。\n",
    "\n",
    "![file](https://ai-studio-static-online.cdn.bcebos.com/1748d860a1114aadbd376c3445ad92b3df26ebc19c6e4a7a8ab39d746ed99f19)\n",
    "\n",
    "同时，`Double DQN`也需要用到两个`Q`网络。`Q1`网络**推荐**能够获得最大`Q`值的动作；`Q2`网络计算这个动作在`Q2`网络中的`Q`值。这时候，`Fixed-Q-Target`的网络结构刚好能派上用场，因此`Double DQN`对`DQN`的唯一的变化就在目标函数上：\n",
    "\n",
    "![file](https://ai-studio-static-online.cdn.bcebos.com/3d76f541bec546aaa822aeb0bcc57b682367bb07967e48e2a1f2a91d7412e330)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## PARL框架DDQN代码分析\n",
    "PARL中`DDQN`的实现可以在`parl/algorithms/fluid/ddqn.py`文件找到"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class DDQN(Algorithm):\n",
    "    def __init__(self, model, act_dim=None, gamma=None, lr=None):\n",
    "        \"\"\" Double DQN algorithm\n",
    "        Args:\n",
    "            model (parl.Model): model defining forward network of Q function | 定义Q函数的前向网络结构\n",
    "            act_dim (int): dimension of the action space | action空间的维度，即有几个action\n",
    "            gamma (float): discounted factor for reward computation. | reward的衰减因子\n",
    "            lr (float): learning rate. | 学习率\n",
    "        \"\"\"\n",
    "        self.model = model\n",
    "        self.target_model = copy.deepcopy(model)\n",
    "\n",
    "        assert isinstance(act_dim, int)\n",
    "        assert isinstance(gamma, float)\n",
    "\n",
    "        self.act_dim = act_dim\n",
    "        self.gamma = gamma\n",
    "        self.lr = lr\n",
    "\n",
    "    def predict(self, obs):\n",
    "        \"\"\" \n",
    "        use value model self.model to predict the action value\n",
    "        使用self.model的value网络来获取 [Q(s,a1),Q(s,a2),...]\n",
    "        \"\"\"\n",
    "        return self.model.value(obs)\n",
    "\n",
    "    def learn(self,\n",
    "              obs,\n",
    "              action,\n",
    "              reward,\n",
    "              next_obs,\n",
    "              terminal,\n",
    "              learning_rate=None):\n",
    "        \"\"\" \n",
    "        update value model self.model with DQN algorithm\n",
    "        使用DQN算法更新self.model的value网络\n",
    "        \"\"\"\n",
    "        # Support the modification of learning_rate\n",
    "        if learning_rate is None:\n",
    "            assert isinstance(\n",
    "                self.lr,\n",
    "                float), \"Please set the learning rate of DQN in initializaion.\"\n",
    "            learning_rate = self.lr\n",
    "\n",
    "        pred_value = self.model.value(obs)\n",
    "        action_onehot = layers.one_hot(action, self.act_dim)\n",
    "        action_onehot = layers.cast(action_onehot, dtype='float32')\n",
    "        pred_action_value = layers.reduce_sum(\n",
    "            layers.elementwise_mul(action_onehot, pred_value), dim=1)\n",
    "\n",
    "        # calculate the target q value\n",
    "        # 计算target网络的q值，这里是和DQN有差异的地方\n",
    "        next_action_value = self.model.value(next_obs)\n",
    "        greedy_action = layers.argmax(next_action_value, axis=-1)\n",
    "        greedy_action = layers.unsqueeze(greedy_action, axes=[1])\n",
    "        greedy_action_onehot = layers.one_hot(greedy_action, self.act_dim)\n",
    "        next_pred_value = self.target_model.value(next_obs)\n",
    "        max_v = layers.reduce_sum(\n",
    "            greedy_action_onehot * next_pred_value, dim=1)\n",
    "        max_v.stop_gradient = True\n",
    "\n",
    "        target = reward + (\n",
    "            1.0 - layers.cast(terminal, dtype='float32')) * self.gamma * max_v\n",
    "        cost = layers.square_error_cost(pred_action_value, target)\n",
    "        cost = layers.reduce_mean(cost)\n",
    "        optimizer = fluid.optimizer.Adam(\n",
    "            learning_rate=learning_rate, epsilon=1e-3)\n",
    "        optimizer.minimize(cost)\n",
    "        return cost\n",
    "\n",
    "    def sync_target(self):\n",
    "        \"\"\" \n",
    "        sync weights of self.model to self.target_model\n",
    "        把 self.model 的模型参数值同步到 self.target_model\n",
    "        \"\"\"\n",
    "        self.model.sync_weights_to(self.target_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## DDQN解决Pong"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd PARL/examples/DQN_variant && python -u train.py --rom ./rom_files/pong.bin --algo DDQN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Dueling DQN\n",
    "\n",
    "`Dueling DQN`是一种网络结构上的创新，其将网络分成了两部分：**值函数** 和 **优势函数**。\n",
    "\n",
    "![file](https://ai-studio-static-online.cdn.bcebos.com/ff6e0be8eb2e484b8961691a57c7a5d856472576d4524fd4aa969d7f0e43b0ed)\n",
    "\n",
    "\n",
    "图中将原有的`DQN`算法的网络输出分成了两部分，在数学上表示为：\n",
    "\n",
    "![file](https://www.zhihu.com/equation?tex=Q%28s%2C+a+%3B+%5Ctheta%2C+%5Calpha%2C+%5Cbeta%29%3DV%28s+%3B+%5Ctheta%2C+%5Cbeta%29%2BA%28s%2C+a+%3B+%5Ctheta%2C+%5Calpha%29+%5C%5C)\n",
    "\n",
    "其中，![file](https://www.zhihu.com/equation?tex=%5Ctheta) 表示网络结构，![file](https://www.zhihu.com/equation?tex=%5Calpha),![file](https://www.zhihu.com/equation?tex=%5Cbeta)表示两个全连接层网络的参数，由图和公式可知，![file](https://www.zhihu.com/equation?tex=V)仅与状态有关，而![file](https://www.zhihu.com/equation?tex=A)与状态和动作都有关。 如果仅仅用当前的这个公式更新的话，其存在一个“**unidentifiable**”问题（比如`V`和`A`分别加上和减去一个值能够得到同样的`Q`，但反过来显然无法由`Q`得到唯一的`V`和`A`）。作者为了解决它，作者强制优势函数估计量在选定的动作处具有**零优势**。 也就是说让网络的最后一个模块实现前向映射，表示为：\n",
    "\n",
    "![file](https://www.zhihu.com/equation?tex=%5Cbegin%7Barray%7D%7Bl%7DQ%28s%2C+a+%3B+%5Ctheta%2C+%5Calpha%2C+%5Cbeta%29%3DV%28s+%3B+%5Ctheta%2C+%5Cbeta%29%2B+%5Cleft%28A%28s%2C+a+%3B+%5Ctheta%2C+%5Calpha%29-%5Cmax+_%7Ba%5E%7B%5Cprime%7D+%5Cin%7C%5Cmathcal%7BA%7D%7C%7D+A%5Cleft%28s%2C+a%5E%7B%5Cprime%7D+%3B+%5Ctheta%2C+%5Calpha%5Cright%29%5Cright%29%5Cend%7Barray%7D+%5C%5C)\n",
    "\n",
    "怎么理解呢？对于任意 ![file](https://www.zhihu.com/equation?tex=a)来说，\n",
    "\n",
    "![file](https://www.zhihu.com/equation?tex=a%5E%7B%2A%7D%3D%5Carg+%5Cmax+_%7Ba%5E%7B%5Cprime%7D+%5Cin+%5Cmathcal%7BA%7D%7D+Q%5Cleft%28s%2C+a%5E%7B%5Cprime%7D+%3B+%5Ctheta%2C+%5Calpha%2C+%5Cbeta%5Cright%29%3D+%5Carg+%5Cmax+_%7Ba%5E%7B%5Cprime%7D+%5Cin+%5Cmathcal%7BA%7D%7D+A%5Cleft%28s%2C+a%5E%7B%5Cprime%7D+%3B+%5Ctheta%2C+%5Calpha%5Cright%29+%5C%5C)\n",
    "\n",
    "那么我们可以得到： ![file](https://www.zhihu.com/equation?tex=Q%5Cleft%28s%2C+a%5E%7B%2A%7D+%3B+%5Ctheta%2C+%5Calpha%2C+%5Cbeta%5Cright%29%3D+V%28s+%3B+%5Ctheta%2C+%5Cbeta%29)，因此，![file](https://www.zhihu.com/equation?tex=V%28s%3B+%5Ctheta%2C+%5Cbeta%29)提供了价值函数的估计，而另一个产生了优势函数的估计。在这里作者使用里**平均**（![file](https://www.zhihu.com/equation?tex=%5Cfrac%7B1%7D%7B%7C%5Cmathcal%7BA%7D%7C%7D)）代替了最大化操作，表示为：\n",
    "\n",
    "![file](https://www.zhihu.com/equation?tex=%5Cbegin%7Barray%7D%7Bl%7DQ%28s%2C+a+%3B+%5Ctheta%2C+%5Calpha%2C+%5Cbeta%29%3DV%28s+%3B+%5Ctheta%2C+%5Cbeta%29%2B+%5Cleft%28A%28s%2C+a+%3B+%5Ctheta%2C+%5Calpha%29-%5Cfrac%7B1%7D%7B%7C%5Cmathcal%7BA%7D%7C%7D+%5Csum_%7Ba%5E%7B%5Cprime%7D%7D+A%5Cleft%28s%2C+a%5E%7B%5Cprime%7D+%3B+%5Ctheta%2C+%5Calpha%5Cright%29%5Cright%29%5Cend%7Barray%7D+%5C%5C)\n",
    "\n",
    "采用这种方法，虽然使得值函数`V`和优势函数`A`不再完美的表示值函数和优势函数(在语义上的表示)，但是这种操作提高了稳定性。而且，并没有改变值函数`V`和优势函数`A`的本质表示。\n",
    "\n",
    "![file](https://ai-studio-static-online.cdn.bcebos.com/212f7132bb4643699062b41ec2537afcc720ec6bcfb04e93b014651c185d1351)\n",
    "\n",
    "在更新网络时，让模型更倾向于去更新`V`而不是更新`A`，这是一种比较有效率的做法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Dueling DQN代码分析\n",
    "`Dueling DQN`是在神经网络结构上的改动，在`DQN_variant`的`Pong`示例中，可在`examples/DQN_variant/atari_model.py`查看`Dueling DQN`的实现代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class AtariModel(parl.Model):\n",
    "    def __init__(self, act_dim, algo='DQN'):\n",
    "        self.act_dim = act_dim\n",
    "\n",
    "        self.conv1 = layers.conv2d(\n",
    "            num_filters=32, filter_size=5, stride=1, padding=2, act='relu')\n",
    "        self.conv2 = layers.conv2d(\n",
    "            num_filters=32, filter_size=5, stride=1, padding=2, act='relu')\n",
    "        self.conv3 = layers.conv2d(\n",
    "            num_filters=64, filter_size=4, stride=1, padding=1, act='relu')\n",
    "        self.conv4 = layers.conv2d(\n",
    "            num_filters=64, filter_size=3, stride=1, padding=1, act='relu')\n",
    "\n",
    "        self.algo = algo\n",
    "        if algo == 'Dueling':\n",
    "            self.fc1_adv = layers.fc(size=512, act='relu')\n",
    "            self.fc2_adv = layers.fc(size=act_dim)\n",
    "            self.fc1_val = layers.fc(size=512, act='relu')\n",
    "            self.fc2_val = layers.fc(size=1)\n",
    "        else:\n",
    "            self.fc1 = layers.fc(size=act_dim)\n",
    "\n",
    "    def value(self, obs):\n",
    "        obs = obs / 255.0\n",
    "        out = self.conv1(obs)\n",
    "        out = layers.pool2d(\n",
    "            input=out, pool_size=2, pool_stride=2, pool_type='max')\n",
    "        out = self.conv2(out)\n",
    "        out = layers.pool2d(\n",
    "            input=out, pool_size=2, pool_stride=2, pool_type='max')\n",
    "        out = self.conv3(out)\n",
    "        out = layers.pool2d(\n",
    "            input=out, pool_size=2, pool_stride=2, pool_type='max')\n",
    "        out = self.conv4(out)\n",
    "        out = layers.flatten(out, axis=1)\n",
    "\n",
    "        if self.algo == 'Dueling':\n",
    "            As = self.fc2_adv(self.fc1_adv(out))\n",
    "            V = self.fc2_val(self.fc1_val(out))\n",
    "            Q = As + (V - layers.reduce_mean(As, dim=1, keep_dim=True))\n",
    "        else:\n",
    "            Q = self.fc1(out)\n",
    "        return Q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Dueling DQN解决Pong"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd PARL/examples/DQN_variant && python -u train.py --rom ./rom_files/pong.bin --algo Dueling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#  PyGame强化学习环境与Catcher小游戏\n",
    "\n",
    "PyGame Learning Environment (PLE)是一个强化学习简单的游戏环境，旨在使开发者可以专注于模型和实验的设计，而不是环境设计。\n",
    "\n",
    "[项目文档](http://pygame-learning-environment.readthedocs.org/)\n",
    "\n",
    "由于PLE中的游戏需要调用可视化窗口，因此训练过程需要在本地进行。\n",
    "\n",
    "## 本地PARL训练环境准备\n",
    "\n",
    "- Python 2.7 or 3.5+. (Windows系统目前仅支持python3.6+以上的环境）\n",
    "- paddlepaddle=1.6.3 \n",
    "- parl==1.3.1\n",
    "\n",
    "## PLE环境安装\n",
    "\n",
    "```bash\n",
    "git clone https://github.com/ntasfi/PyGame-Learning-Environment.git \n",
    "# 可用镜像替代https://gitee.com/cqh1_admin/PyGame-Learning-Environment.git\n",
    "cd PyGame-Learning-Environment/\n",
    "pip install -e \n",
    "```\n",
    "\n",
    "## Catcher游戏设置\n",
    "\n",
    "PLE中Catcher小游戏的初始化方式如下：\n",
    "\n",
    "```python\n",
    "from ple.games.catcher import Catcher\n",
    "# 默认64*64的游戏窗口过小，将其调整到256*256\n",
    "env = Catcher(width=256, height=256)\n",
    "# 显示游戏画面\n",
    "p = PLE(env, fps=30, display_screen=True, force_fps=True)\n",
    "```\n",
    "\n",
    "实际在本地运行时，如果发现接发球的游戏画面速度过快，可以到`ple/games/catcher.py`中调整游戏速度\n",
    "\n",
    "```python\n",
    "class Catcher(base.PyGameWrapper):\n",
    "\n",
    "    def __init__(self, width=64, height=64, init_lives=3):\n",
    "\n",
    "        actions = {\n",
    "            \"left\": K_a,\n",
    "            \"right\": K_d\n",
    "        }\n",
    "\n",
    "        base.PyGameWrapper.__init__(self, width, height, actions=actions)\n",
    "\n",
    "        self.fruit_size = percent_round_int(height, 0.06)\n",
    "        # 落球速度\n",
    "        self.fruit_fall_speed = 0.0004 * height\n",
    "        # 接球速度\n",
    "        self.player_speed = 0.002 * width\n",
    "        ……\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 搭建Model\n",
    "实现`Dueling DQN`的网络结构"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Model(parl.Model):\n",
    "    def __init__(self, act_dim):\n",
    "\n",
    "        self.fc1_adv = layers.fc(size=128, act='relu')\n",
    "        self.fc2_adv = layers.fc(size=128, act='relu')\n",
    "        self.fc3_adv = layers.fc(size=act_dim)\n",
    "        self.fc1_val = layers.fc(size=128, act='relu')\n",
    "        self.fc2_val = layers.fc(size=128, act='relu')\n",
    "        self.fc3_val = layers.fc(size=1)\n",
    "\n",
    "    def value(self, obs):\n",
    "        # 定义网络\n",
    "        # 输入state，输出所有action对应的Q，[Q(s,a1), Q(s,a2), Q(s,a3)...]\n",
    "\n",
    "        As = self.fc3_adv(self.fc2_adv(self.fc1_adv(obs)))\n",
    "        V = self.fc3_val(self.fc2_val(self.fc1_val(obs)))\n",
    "        Q = As + (V - layers.reduce_mean(As, dim=1, keep_dim=True))\n",
    "\n",
    "        return Q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 搭建Agent\n",
    "- `Agent` 负责算法与环境的交互，在交互过程中把生成的数据提供给`Algorithm`来更新模型(`Model`)，数据的预处理流程也一般定义在这里。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Agent(parl.Agent):\n",
    "    def __init__(self,\n",
    "                 algorithm,\n",
    "                 obs_dim,\n",
    "                 act_dim,\n",
    "                 e_greed=0.1,\n",
    "                 e_greed_decrement=0):\n",
    "        assert isinstance(obs_dim, int)\n",
    "        assert isinstance(act_dim, int)\n",
    "        self.obs_dim = obs_dim\n",
    "        self.act_dim = act_dim\n",
    "        super(Agent, self).__init__(algorithm)\n",
    "\n",
    "        self.global_step = 0\n",
    "        self.update_target_steps = 200 # 每隔200个training steps再把model的参数复制到target_model中\n",
    "\n",
    "        self.e_greed = e_greed  # 有一定概率随机选取动作，探索\n",
    "        self.e_greed_decrement = e_greed_decrement  # 随着训练逐步收敛，探索的程度慢慢降低\n",
    "\n",
    "    def build_program(self):\n",
    "        self.pred_program = fluid.Program()\n",
    "        self.learn_program = fluid.Program()\n",
    "\n",
    "        with fluid.program_guard(self.pred_program):  # 搭建计算图用于预测动作，定义输入输出变量\n",
    "            obs = layers.data(\n",
    "                name='obs', shape=[self.obs_dim], dtype='float32')\n",
    "            self.value = self.alg.predict(obs)\n",
    "\n",
    "        with fluid.program_guard(self.learn_program):  # 搭建计算图用于更新Q网络，定义输入输出变量\n",
    "            obs = layers.data(\n",
    "                name='obs', shape=[self.obs_dim], dtype='float32')\n",
    "            action = layers.data(name='act', shape=[1], dtype='int32')\n",
    "            reward = layers.data(name='reward', shape=[], dtype='float32')\n",
    "            next_obs = layers.data(\n",
    "                name='next_obs', shape=[self.obs_dim], dtype='float32')\n",
    "            terminal = layers.data(name='terminal', shape=[], dtype='bool')\n",
    "            self.cost = self.alg.learn(obs, action, reward, next_obs, terminal)\n",
    "\n",
    "    def sample(self, obs):\n",
    "        sample = np.random.rand()  # 产生0~1之间的小数\n",
    "        if sample < self.e_greed:\n",
    "            act = np.random.randint(self.act_dim)  # 探索：每个动作都有概率被选择\n",
    "        else:\n",
    "            act = self.predict(obs)  # 选择最优动作\n",
    "        self.e_greed = max(\n",
    "            0.01, self.e_greed - self.e_greed_decrement)  # 随着训练逐步收敛，探索的程度慢慢降低\n",
    "        return act\n",
    "\n",
    "    def predict(self, obs):  # 选择最优动作\n",
    "        obs = np.expand_dims(obs, axis=0)\n",
    "        pred_Q = self.fluid_executor.run(\n",
    "            self.pred_program,\n",
    "            feed={'obs': obs.astype('float32')},\n",
    "            fetch_list=[self.value])[0]\n",
    "        pred_Q = np.squeeze(pred_Q, axis=0)\n",
    "        act = np.argmax(pred_Q)  # 选择Q最大的下标，即对应的动作\n",
    "        return act\n",
    "\n",
    "    def learn(self, obs, act, reward, next_obs, terminal):\n",
    "        # 每隔200个training steps同步一次model和target_model的参数\n",
    "        if self.global_step % self.update_target_steps == 0:\n",
    "            self.alg.sync_target()\n",
    "        self.global_step += 1\n",
    "\n",
    "        act = np.expand_dims(act, -1)\n",
    "        feed = {\n",
    "            'obs': obs.astype('float32'),\n",
    "            'act': act.astype('int32'),\n",
    "            'reward': reward,\n",
    "            'next_obs': next_obs.astype('float32'),\n",
    "            'terminal': terminal\n",
    "        }\n",
    "        cost = self.fluid_executor.run(\n",
    "            self.learn_program, feed=feed, fetch_list=[self.cost])[0]  # 训练一次网络\n",
    "        return cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 搭建Algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from parl.algorithms import DDQN # 直接从parl库中导入DDQN算法，无需自己重写算法\n",
    "# 根据parl框架构建agent\n",
    "model = Model(act_dim=action_dim)\n",
    "algorithm = DDQN(model, act_dim=action_dim, gamma=GAMMA, lr=LEARNING_RATE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## ReplayMemory\n",
    "- 经验池：用于存储多条经验，实现经验回放。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class ReplayMemory(object):\n",
    "    def __init__(self, max_size):\n",
    "        self.buffer = collections.deque(maxlen=max_size)\n",
    "\n",
    "    # 增加一条经验到经验池中\n",
    "    def append(self, exp):\n",
    "        self.buffer.append(exp)\n",
    "\n",
    "    # 从经验池中选取N条经验出来\n",
    "    def sample(self, batch_size):\n",
    "        mini_batch = random.sample(self.buffer, batch_size)\n",
    "        obs_batch, action_batch, reward_batch, next_obs_batch, done_batch = [], [], [], [], []\n",
    "\n",
    "        for experience in mini_batch:\n",
    "            s, a, r, s_p, done = experience\n",
    "            obs_batch.append(s)\n",
    "            action_batch.append(a)\n",
    "            reward_batch.append(r)\n",
    "            next_obs_batch.append(s_p)\n",
    "            done_batch.append(done)\n",
    "\n",
    "        return np.array(obs_batch).astype('float32'), \\\n",
    "            np.array(action_batch).astype('float32'), np.array(reward_batch).astype('float32'),\\\n",
    "            np.array(next_obs_batch).astype('float32'), np.array(done_batch).astype('float32')\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.buffer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 本地完整可执行代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from ple.games.catcher import Catcher\n",
    "from ple import PLE\n",
    "import parl\n",
    "from parl import layers\n",
    "import paddle.fluid as fluid\n",
    "import numpy as np\n",
    "import os\n",
    "from parl.utils import logger\n",
    "import random\n",
    "import collections\n",
    "from parl.algorithms import DDQN # 直接从parl库中导入DDQN算法，无需自己重写算法\n",
    "import random\n",
    "import collections\n",
    "import numpy as np\n",
    "import pygame\n",
    "\n",
    "LEARN_FREQ = 5 # 训练频率，不需要每一个step都learn，攒一些新增经验后再learn，提高效率\n",
    "MEMORY_SIZE = 80000    # replay memory的大小，越大越占用内存\n",
    "MEMORY_WARMUP_SIZE = 2048 # replay_memory 里需要预存一些经验数据，再从里面sample一个batch的经验让agent去learn\n",
    "BATCH_SIZE = 1024   # 每次给agent learn的数据数量，从replay memory随机里sample一批数据出来\n",
    "GAMMA = 0.9 # reward 的衰减因子，一般取 0.9 到 0.999 不等\n",
    "\n",
    "LEARNING_RATE = 0.01 # 学习率\n",
    "\n",
    "\n",
    "class Model(parl.Model):\n",
    "    def __init__(self, act_dim):\n",
    "\n",
    "        self.fc1_adv = layers.fc(size=128, act='relu')\n",
    "        self.fc2_adv = layers.fc(size=128, act='relu')\n",
    "        self.fc3_adv = layers.fc(size=act_dim)\n",
    "        self.fc1_val = layers.fc(size=128, act='relu')\n",
    "        self.fc2_val = layers.fc(size=128, act='relu')\n",
    "        self.fc3_val = layers.fc(size=1)\n",
    "\n",
    "    def value(self, obs):\n",
    "        # 定义网络\n",
    "        # 输入state，输出所有action对应的Q，[Q(s,a1), Q(s,a2), Q(s,a3)...]\n",
    "\n",
    "        As = self.fc3_adv(self.fc2_adv(self.fc1_adv(obs)))\n",
    "        V = self.fc3_val(self.fc2_val(self.fc1_val(obs)))\n",
    "        Q = As + (V - layers.reduce_mean(As, dim=1, keep_dim=True))\n",
    "\n",
    "        return Q\n",
    "\n",
    "\n",
    "class Agent(parl.Agent):\n",
    "    def __init__(self,\n",
    "                 algorithm,\n",
    "                 obs_dim,\n",
    "                 act_dim,\n",
    "                 e_greed=0.1,\n",
    "                 e_greed_decrement=0):\n",
    "        assert isinstance(obs_dim, int)\n",
    "        assert isinstance(act_dim, int)\n",
    "        self.obs_dim = obs_dim\n",
    "        self.act_dim = act_dim\n",
    "        super(Agent, self).__init__(algorithm)\n",
    "\n",
    "        self.global_step = 0\n",
    "        self.update_target_steps = 200 # 每隔200个training steps再把model的参数复制到target_model中\n",
    "\n",
    "        self.e_greed = e_greed  # 有一定概率随机选取动作，探索\n",
    "        self.e_greed_decrement = e_greed_decrement  # 随着训练逐步收敛，探索的程度慢慢降低\n",
    "\n",
    "    def build_program(self):\n",
    "        self.pred_program = fluid.Program()\n",
    "        self.learn_program = fluid.Program()\n",
    "\n",
    "        with fluid.program_guard(self.pred_program):  # 搭建计算图用于 预测动作，定义输入输出变量\n",
    "            obs = layers.data(\n",
    "                name='obs', shape=[self.obs_dim], dtype='float32')\n",
    "            self.value = self.alg.predict(obs)\n",
    "\n",
    "        with fluid.program_guard(self.learn_program):  # 搭建计算图用于 更新Q网络，定义输入输出变量\n",
    "            obs = layers.data(\n",
    "                name='obs', shape=[self.obs_dim], dtype='float32')\n",
    "            action = layers.data(name='act', shape=[1], dtype='int32')\n",
    "            reward = layers.data(name='reward', shape=[], dtype='float32')\n",
    "            next_obs = layers.data(\n",
    "                name='next_obs', shape=[self.obs_dim], dtype='float32')\n",
    "            terminal = layers.data(name='terminal', shape=[], dtype='bool')\n",
    "            self.cost = self.alg.learn(obs, action, reward, next_obs, terminal)\n",
    "\n",
    "    def sample(self, obs):\n",
    "        sample = np.random.rand()  # 产生0~1之间的小数\n",
    "        if sample < self.e_greed:\n",
    "            act = np.random.randint(self.act_dim)  # 探索：每个动作都有概率被选择\n",
    "        else:\n",
    "            act = self.predict(obs)  # 选择最优动作\n",
    "        self.e_greed = max(\n",
    "            0.01, self.e_greed - self.e_greed_decrement)  # 随着训练逐步收敛，探索的程度慢慢降低\n",
    "        return act\n",
    "\n",
    "    def predict(self, obs):  # 选择最优动作\n",
    "        obs = np.expand_dims(obs, axis=0)\n",
    "        pred_Q = self.fluid_executor.run(\n",
    "            self.pred_program,\n",
    "            feed={'obs': obs.astype('float32')},\n",
    "            fetch_list=[self.value])[0]\n",
    "        pred_Q = np.squeeze(pred_Q, axis=0)\n",
    "        act = np.argmax(pred_Q)  # 选择Q最大的下标，即对应的动作\n",
    "        return act\n",
    "\n",
    "    def learn(self, obs, act, reward, next_obs, terminal):\n",
    "        # 每隔200个training steps同步一次model和target_model的参数\n",
    "        if self.global_step % self.update_target_steps == 0:\n",
    "            self.alg.sync_target()\n",
    "        self.global_step += 1\n",
    "\n",
    "        act = np.expand_dims(act, -1)\n",
    "        feed = {\n",
    "            'obs': obs.astype('float32'),\n",
    "            'act': act.astype('int32'),\n",
    "            'reward': reward,\n",
    "            'next_obs': next_obs.astype('float32'),\n",
    "            'terminal': terminal\n",
    "        }\n",
    "        cost = self.fluid_executor.run(\n",
    "            self.learn_program, feed=feed, fetch_list=[self.cost])[0]  # 训练一次网络\n",
    "        return cost\n",
    "\n",
    "\n",
    "class ReplayMemory(object):\n",
    "    def __init__(self, max_size):\n",
    "        self.buffer = collections.deque(maxlen=max_size)\n",
    "\n",
    "    # 增加一条经验到经验池中\n",
    "    def append(self, exp):\n",
    "        self.buffer.append(exp)\n",
    "\n",
    "    # 从经验池中选取N条经验出来\n",
    "    def sample(self, batch_size):\n",
    "        mini_batch = random.sample(self.buffer, batch_size)\n",
    "        obs_batch, action_batch, reward_batch, next_obs_batch, done_batch = [], [], [], [], []\n",
    "\n",
    "        for experience in mini_batch:\n",
    "            s, a, r, s_p, done = experience\n",
    "            obs_batch.append(s)\n",
    "            action_batch.append(a)\n",
    "            reward_batch.append(r)\n",
    "            next_obs_batch.append(s_p)\n",
    "            done_batch.append(done)\n",
    "\n",
    "        return np.array(obs_batch).astype('float32'), \\\n",
    "            np.array(action_batch).astype('float32'), np.array(reward_batch).astype('float32'),\\\n",
    "            np.array(next_obs_batch).astype('float32'), np.array(done_batch).astype('float32')\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.buffer)\n",
    "\n",
    "\n",
    "# 训练一个episode\n",
    "def run_episode(p, agent, rpm):\n",
    "    total_reward = 0\n",
    "    p.reset_game()\n",
    "    obs = list(p.getGameState().values())\n",
    "    step = 0\n",
    "    while True:\n",
    "        step += 1\n",
    "        action_index = agent.sample(obs)\n",
    "        action = p.getActionSet()[action_index]  # 采样动作，所有动作都有概率被尝试到\n",
    "\n",
    "        # 行动\n",
    "        next_obs, reward, done = list(p.getGameState().values()), p.act(action), p.game_over()\n",
    "        rpm.append((obs, action, reward, next_obs, done))\n",
    "\n",
    "        # train model\n",
    "        if (len(rpm) > MEMORY_WARMUP_SIZE) and (step % LEARN_FREQ == 0):\n",
    "            (batch_obs, batch_action, batch_reward, batch_next_obs,\n",
    "             batch_done) = rpm.sample(BATCH_SIZE)\n",
    "            train_loss = agent.learn(batch_obs, batch_action, batch_reward,\n",
    "                                     batch_next_obs,\n",
    "                                     batch_done)  # s,a,r,s',done\n",
    "\n",
    "        total_reward += reward\n",
    "        obs = next_obs\n",
    "        if done:\n",
    "            break\n",
    "    return total_reward\n",
    "\n",
    "\n",
    "# 评估 agent, 跑 5 个episode，总reward求平均\n",
    "def evaluate(p, agent, render=False):\n",
    "    eval_reward = []\n",
    "    for i in range(5):\n",
    "        p.reset_game()\n",
    "        obs = list(p.getGameState().values())\n",
    "        episode_reward = 0\n",
    "        while True:\n",
    "            action_index = agent.sample(obs)\n",
    "            action = p.getActionSet()[action_index]  # 采样动作，所有动作都有概率被尝试到\n",
    "            # 行动\n",
    "            next_obs, reward = list(p.getGameState().values()), p.act(action)\n",
    "            episode_reward += reward\n",
    "            if render:\n",
    "                p.getScreenRGB()\n",
    "            if p.game_over():\n",
    "                break\n",
    "        eval_reward.append(episode_reward)\n",
    "    return np.mean(eval_reward)\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "\n",
    "    # 创建环境\n",
    "    env = Catcher(width=256, height=256)\n",
    "    p = PLE(env, fps=30, display_screen=True, force_fps=True)\n",
    "    action_dim = len(p.getActionSet())\n",
    "    obs_shape = len(p.getGameState())\n",
    "\n",
    "    # 创建经验池\n",
    "    rpm = ReplayMemory(MEMORY_SIZE)  # DQN的经验回放池\n",
    "\n",
    "    # 根据parl框架构建agent\n",
    "    model = Model(act_dim=action_dim)\n",
    "    algorithm = DDQN(model, act_dim=action_dim, gamma=GAMMA, lr=LEARNING_RATE)\n",
    "    agent = Agent(\n",
    "        algorithm,\n",
    "        obs_dim=obs_shape,\n",
    "        act_dim=action_dim,\n",
    "        e_greed=0.1,  # 有一定概率随机选取动作，探索\n",
    "        e_greed_decrement=100000)  # 随着训练逐步收敛，探索的程度慢慢降低\n",
    "\n",
    "    # # 加载模型\n",
    "    # save_path = 'model_dir/steps_5000.ckpt'\n",
    "    # agent.restore(save_path)\n",
    "\n",
    "    # 先往经验池里存一些数据，避免最开始训练的时候样本丰富度不够\n",
    "    while len(rpm) < MEMORY_WARMUP_SIZE:\n",
    "        run_episode(p, agent, rpm)\n",
    "\n",
    "    max_episode = 50000\n",
    "\n",
    "    # 开始训练\n",
    "    episode = 0\n",
    "    while episode < max_episode:  # 训练max_episode个回合，test部分不计算入episode数量\n",
    "        # train part\n",
    "        for i in range(0, 50):\n",
    "            total_reward = run_episode(p, agent, rpm)\n",
    "            # print(total_reward)\n",
    "            episode += 1\n",
    "\n",
    "        # test part\n",
    "        eval_reward = evaluate(p, agent, render=True)  # render=True 查看显示效果\n",
    "        logger.info('episode:{}    e_greed:{}   test_reward:{}'.format(\n",
    "            episode, agent.e_greed, eval_reward))\n",
    "\n",
    "        # 训练结束，保存模型\n",
    "        ckpt = 'model_dir/steps_{}.ckpt'.format(episode)\n",
    "        agent.save(ckpt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 训练过程\n",
    "\n",
    "<img src=\"https://ai-studio-static-online.cdn.bcebos.com/e751e569497b46dcb8e35613cb004337dc8034ee4aae4c84b9c52e8ebaf07fb6\"  style=\"zoom: 345%;\" />\n",
    "\n",
    "![file](https://ai-studio-static-online.cdn.bcebos.com/8e9c5038823a4dca881bd1d996d05900fce60d31295d47309644bc4eee3dae4c)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 参考资料\n",
    "\n",
    "- DQN in [Human-level Control Through Deep Reinforcement Learning](http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html)\n",
    "\n",
    "- DDQN in [Deep Reinforcement Learning with Double Q-learning](https://arxiv.org/abs/1509.06461)\n",
    "\n",
    "- Dueling DQN in [Dueling Network Architectures for Deep Reinforcement Learning](https://arxiv.org/abs/1511.06581)\n",
    "\n",
    "- [李宏毅 强化学习 DQN](https://www.bilibili.com/video/BV11x411f7qt?from=search&seid=12688169522000021540)\n",
    "\n",
    "- [DeepRL系列(10): Dueling DQN(DDQN)原理及实现](https://zhuanlan.zhihu.com/p/114834834)\n",
    "\n",
    "- [Deep Q Network](https://morvanzhou.github.io/tutorials/machine-learning/reinforcement-learning/)\n",
    "\n",
    "- [强化学习—DQN算法原理详解](https://wanjun0511.github.io/2017/11/05/DQN/)\n",
    "\n",
    "- [DQN三大改进(一)-Double DQN](https://www.jianshu.com/p/fae51b5fe000)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PaddlePaddle 1.6.2 (Python 3.5)",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
